Papers with model comparison
Validating Automatic Evaluation of Controllable Counterspeech Generation: Rankings Matter More Than Scores (2026.eacl-long)
Copied to clipboard
| Challenge: | Existing methods for evaluating attributes of counterspeech are limited and the validity of such evaluations is questionable when the classifiers themselves have only modest performance. |
| Approach: | They examine the automatic evaluation of counterspeech attributes using a multi-attribute counterseech dataset containing 2,728 samples. |
| Outcome: | The proposed model can be trusted by classifier validation, and it can rank models with confidence. |
Follow the leader(board) with confidence: Estimating p-values from a single test set with item and response variance (2023.findings-acl)
Copied to clipboard
| Challenge: | Among the problems with leaderboard culture in NLP has been the widespread lack of confidence estimation in reported results. |
| Approach: | They propose a framework and simulator for estimating p-values for comparisons between the results of two systems using variance found naturally (though rarely reported) in test set items and individual labels on an item (responses). |
| Outcome: | The proposed framework and simulator are used to estimate p-values for comparisons between the results of two systems under the assumption that the null hypothesis is true. |
Bayes Test of Precision, Recall, and F1 Measure for Comparison of Two Natural Language Processing Models (P19-1)
Copied to clipboard
| Challenge: | Existing t-tests for cross-validation (CV) are inappropriate for model comparison . existing t tests for cross validation (CV), such as 52 CV t test and F ttest, are inadequate . |
| Approach: | They propose to use a block-regularized 32 CV to compare two NLP models . they calibrate the posterior distributions of P, R, and F1 and derive an accurate interval estimation of P and R . |
| Outcome: | The proposed model could regularize the difference in certain frequency distributions over linguistic units and yield stable estimators of P, R, and F1. |
Putting HUMANS first: Efficient LAM Evaluation with Human Preference Alignment (2026.acl-long)
Copied to clipboard
| Challenge: | Existing LAM benchmarks with thousands of examples create substantial computational barriers. |
| Approach: | They examine whether subsets can reliably evaluate large audio models . they find that subset of 50 examples can achieve over 0.93 Pearson correlation with full benchmark . |
| Outcome: | The proposed method outperforms the full benchmark and subset selection methods. |